Repository navigation
Horizon: let the template fork workers for supervisors inside the master - #7
Conversation
A unit's fork server forks, admits, signals and reaps the horizon:work workers of its supervisor loop. That handling moves into WorkerSet, unchanged, so another server can fork workers the same way. UnitServer keeps the loop and passes it every worker message, as before.
Some apps run Horizon's supervisors inside the master process instead of one process each. Their workers can now be forked as well: the master asks its template for each worker (a spawn with role: worker), and the template forks it through WorkerSet, with no fork server in between. The template TERMs the workers it forked when its master is gone, so none outlives it. On the master's side, Master::start() in inline mode (queue-prefork.horizon_inline) skips the check for the app's own AddSupervisor binding, applies the off switches itself and pumps on MasterSupervisorLooped. Master::workerProcess() hands out forked handles with a crash breaker per supervisor, and ForksWorkersFromTemplate lets the app's process pool use them. Horizon restarts a worker on the handle it has, so a stopped handle whose template is gone, or whose supervisor the breaker turned stock, is replaced with a new one, forked again or stock. The doctor reads horizon_inline as well.
|
Hi Volodymyr, thanks for the PR and the detailed write-up! To be clear up front, I'm not against the change. It's careful work: it reuses the fork server's worker handling, setups that don't opt in are untouched, and the tests pass. Before merging, though, I'd like to understand what keeps you on inline supervisors, because once queue-prefork is on, most of what they save is already gone. Rough numbers from the README benchmarks, for 10 supervisors and 40 workers (your app's sizes will differ, but the ratios should hold):
A forked supervisor costs about 6 MB instead of ~56 MB, so on top of queue-prefork, running them inline saves about 6 MB each. I went through something similar myself. Before queue-prefork, my Horizon ran with a trimmed boot that loaded only what Horizon and the workers needed. That gave sizable savings, but with queue-prefork they became a rounding error, so I went back to starting Horizon normally. So is it memory for you, or something else: fewer processes, fewer Redis connections, something about how your supervisors run? If there's a reason to stay inline, I'm happy to take this, with a few small changes on top (mainly falling back to stock when a worker can't be spawned, and an integration test with real Horizon). If it's only memory, standard Horizon with queue-prefork gets you nearly all of it without maintaining your own Horizon subclasses. |
|
Thanks for looking at it so closely, and for the numbers. Your per-supervisor figure holds for us: on top of queue-prefork, a supervisor run inline saves about 7 MB. What decides whether that matters is the ratio of supervisors to workers. At 10 supervisors and 40 workers it's a rounding error. But an idle forked worker costs about as much as a forked supervisor, so when the two counts are close, inline mode removes about half of what's left. That's our case on dev machines, where RAM is what limits us: there we run roughly as many supervisors as workers, and inline mode takes queue-prefork's ~1.2 GB down to ~0.7 GB. At around 100 of each, that's half the memory and half the Redis connections. Each supervisor process holds its own connections (Horizon's, for commands and its repository, plus its queue connection, which the autoscaler reads every loop), while inline supervisors share the master's. The process count drops by more than half, since with queue-prefork each supervisor is a supervisor process plus its fork server. The inline subclasses predate queue-prefork in our app, so keeping them isn't new work for us. Happy to add both changes: falling back to stock when a worker can't be spawned, and an integration test with real Horizon. I'll push them to this branch. |
|
Thanks, that makes sense. With about one worker per supervisor the numbers look very different, and the Redis connections are a good point too. Go ahead and push both changes; I'll add a couple of small things on top, then merge and release it as 1.2.0. |
A worker the template could not fork left its slot empty: Horizon restarted it on the same handle, which asked the template again on every restart. Its supervisor now turns stock (a UnitStock event), and the pool execs a worker in its place. The workbench runs Horizon's supervisors inside the master behind HORIZON_INLINE, and HorizonInlineTest drives real Horizon that way: the process tree, jobs, pause and continue, a killed worker, a failed fork (a process limit on the template) and horizon:terminate. CI runs it as its own Docker leg.
|
Pushed both in b9dbcf3.
Two things the test ran into, both handled in the workbench: Horizon holds back a worker restarted within the last minute, so the fallback test kills a worker no other test touches; and a paused master skips its supervisors' loops, so the workbench persists the inline supervisors on |
|
Thanks Volodymyr, both changes look good, and the integration test is great. GitHub doesn't let me push to an organization's fork, so I opened #8 with your commits unchanged plus a few of mine; merging it marks this one merged too. The main additions: a shared crash breaker, an inline-only compatibility check for what the trait relies on, the package no longer replacing the app's It also turned up a pause bug that affects every Horizon setup, fixed there too. I'll merge once CI is green and release 1.2.0. |
Supervisors inside the Horizon master (#7)
Some apps run Horizon's supervisors inside the master process instead of one process each, to save the memory of a booted app per supervisor. Today such an app has to run queue-prefork stock: its own
AddSupervisorbinding fails the preflight, and nothing could fork its workers anyway, since workers are forked only by a unit's fork server at its supervisor loop's request.This PR lets the template fork those workers directly, on the master's request. In our measurements the two savings add up: the in-process supervisors cost nothing extra, and every worker shares the template's compiled code.
What changes
WorkerSet(first commit, no behaviour change). The fork server's handling ofhorizon:workworkers (fork, admit, pause state, cancel, signal, ready, reap) moves intosrc/Server/WorkerSet.phpso that a second server can use it.UnitServerkeeps its loop and passes it every worker message as before.The template forks workers (second commit).
ForkedProcesssends its role inspawn. Forrole: worker, the template forks ahorizon:workworker through its ownWorkerSet, routesadmit,pause_state,cancelandsignalfor worker ids, and reaps them. Anything else still forks a unit, so existing masters are unaffected.masterExited(), before it exits or execs the stock master; spawned: when the master is gone or its channel closes), so none outlives it. The fork blocks the relayed signals until the child has reset its handlers.Master::start()takes an inline flag, defaulting to the new confighorizon_inline(QUEUE_PREFORK_HORIZON_INLINE). Inline mode skips Preflight's bindings check only, applies the off switches itself (asWiringdoes before binding the package's command), and registers its own tick onMasterSupervisorLooped, since the app's command replacesPreforkHorizonCommand.Master::workerProcess(SupervisorOptions)hands out a forked handle, or null for stock, with the unit breaker's rule kept per supervisor.ForksWorkersFromTemplateis a trait for the app'sProcessPoolsubclass.createProcess()asks the master for a forked handle. Horizon restarts a stopped worker on the handle it has, somonitor()first replaces a stopped forked handle whose template is gone, or whose supervisor the breaker turned stock, with a new one.queue-prefork:doctorchecks this mode whenhorizon_inlineis set.horizon:purgemust not run in this mode); CHANGELOG gets an Unreleased entry.Limitations
AddSupervisor::createProcess()could useunitProcess()for it, but that is left out here.Happy to rename the config key, split the PR, or move the trait wherever it fits better.